1. Field of the Invention
This invention relates to an image processing apparatus, an image processing method and a program, and more particularly to an image processing apparatus, an image processing method and a program by which flexible resolution enhancement of a image can be obtained.
2. Description of the Related Art
For example, in digital cameras in recent years, enhancement of the performance of an image pickup device such as, for example, a CMOS (Complementary Metal Oxide Semiconductor) imager and the performance of image processing have been and are advancing. For example, a digital camera is available which combines a plurality of images as input images picked up successively at a high rate to produce an image of high picture quality. As such images of high picture quality, for example, an image for which camera shake correction is carried out, an image from which noise is removed to enhance the S/N ratio (Signal to Noise ratio), an image of a high resolution having an increased number of pixels and so forth are available.
For example, production of an image having an increased number of pixels from a single image (hereinafter referred to sometimes as image of a frame) can be carried out by interpolation by means of an interpolation low-pass filter whose pass-band is a frequency region lower than the Nyquist frequency and which uses a sinc function (sin(x)/x), by interpolation by a spline function or the like or by some other interpolation.
However, although the interpolation described above can increase the number of pixels, it fails to enhance the resolution.
Meanwhile, the image pickup device of a digital camera sometimes has the Bayer array. In this instance, sampling of, for example, a R component from among R (Red), G (Green) and B (Blue) components is carried out in a state wherein pixels other than pixels of the R component are sampled out or removed. This similarly applies also to the G component and the B component. Accordingly, in an image picked up by the image pickup device of the Bayer array, some pixels are removed from an image of each color component.
Further, for example, in a television receiver, an image of the interlaced type is sometimes handled. An image of the interlaced type (hereinafter referred to sometimes as interlaced image) is considered an image from which pixels in every other line are removed.
The pixel values of such an image from which pixels are removed as described above include aliasing components in addition to signal components. If the interpolation described above is carried out for an image which includes aliasing components, then an image obtained as a result of the interpolation has conspicuous aliasing components.
Thus, a resolution enhancement technique is available wherein a plurality of frames are combined to produce an image of one frame whose pixel number is increased and which has an enhanced resolution. Such a resolution enhancement technique as just described is disclosed, for example, in Japanese Patent Laid-Open No. Hei 09-69755 (hereinafter referred to as Patent Document 1) which corresponds to U.S. Pat. No. 6,023,535, Japanese Patent Laid-Open No. Hei 08-336046 (hereinafter referred to as Patent Document 3), Japanese Patent Laid-Open No. 2007-324789 (hereinafter referred to as Patent Document 3) which corresponds to U.S. Patent Application No. 2008018786 and Japanese Patent Laid-Open No. 2000-216682 (hereinafter referred to as Patent Document 4) which corresponds to U.S. Pat. No. 6,507,859.
In the existing resolution-enhancement technique, enhancement of the resolution of an image is carried out by three processes including a movement detection process, a wideband interpolation process and a weighted addition process.
In the movement detection process, images of a plurality of frames inputted, that is, input images, are used to detect a movement of an image to estimate a difference between positions at or phases in which the images are sampled, that is, between sampling positions which are spatial positions of light received by the pixels of an image pickup device.
Here, the movement detection process can be carried out, for example, by a gradient method, a block matching method and various other methods.
In the wideband interpolation process, a low-pass filter having a wide frequency band for passing all high frequency components of an image including aliasing components is used to interpolate pixels or sampling points to increase the number of pixels so that an image of an enhanced resolution is produced.
The wideband interpolation process can be carried out using a popular wideband LPF (Low Pass Filter) having a passband equal to twice the Nyquist frequency as described, for example, in Patent Document 1 or 2.
In the weighted addition process, mixture, that is, weighted addition, of images of a plurality of frames picked up at a high rate is carried out in accordance with a weight corresponding to the phase of sampling of the images of a plurality of frames. Consequently, aliasing components generated upon sampling of the images are canceled and removed while high frequency components of the images are restored.
It is to be noted that Patent Document 1 discloses a technique of combining nine frames of images having aliasing components two-dimensionally to produce a high resolution image. Patent Document 2 discloses a technique for combining three frames of images having aliasing components one-dimensionally to produce a high resolution image. Patent Document 3 and Patent Document 4 disclose a technique for IP (Interlace Progressive) conversion of combining images of two frames for which Hilbert transform is carried out to produce an image of a high resolution to carry out conversion from an interlaced image to a progressive image.
The existing resolution enhancement technique is described further with reference to FIGS. 1A to 1C.
FIG. 1A illustrates frequency spectra of images of three successive frames (hereinafter referred to as frames #1, #2 and #3) picked up at a high rate and having different sampling phases within a one-dimensional (one spatial direction) frequency region.
In FIG. 1A, the distance from the axis of the frequency represents the signal intensity, and the rotational angle around the axis of the frequency represents the phase.
Where the frames #1 to #3 include aliasing components in addition to signal components, if a wideband interpolation process is carried out for the frame #i (here, i=1, 2 or 3), that is, if interpolation of pixels is carried out by means of a wideband LPF which passes a frequency band from −fs to fs (in FIG. 1A, only a frequency band from 0 to fs is illustrated) equal to twice the Nyquist frequency fs/2, then an image which includes signal components and aliasing components according to the phase of sampling over the frequency band from −fs to fs which is equal to twice the Nyquist frequency fs/2 as seen in FIG. 1A is obtained.
The image obtained by the wideband interpolation process includes a number of pixels increased from that of the original image.
FIG. 1B illustrates a phase of signal components of the frames #1 to #3 obtained by the wideband interpolation process, and FIG. 1C illustrates a phase of aliasing components of the frames #1 to #3 obtained by the wideband interpolation process.
It is to be noted that, in FIGS. 1B and 1C, the axis of abscissa indicates the imaginary axis and the axis of ordinate indicates the real axis.
The phases of signal components of the three frames #1 to #3 obtained by the wideband interpolation process coincide with each other, and, for example, if the phase of aliasing components of the frame #1 is determined as a reference, the phases of aliasing components rotate in response to the difference in phase of sampling from the frame #1.
As described above, the phase of aliasing components of the three frames #1 to #3 obtained by the wideband interpolation process rotates in response to the difference in phase of sampling from the frame #1. Accordingly, the aliasing components of the three frames #1 to #3 obtained by the wideband interpolation process can be removed by carrying out weighted addition of the three frames #1 to #3 with a weight determined in accordance with the difference in phase of sampling from the frame #1.
Therefore, in the existing resolution enhancement technique, a weighted addition process of carrying out weighted addition of the images of the frames #1 to #3 with the weight determined in accordance with the (difference in) phase of sampling of the three frames #1 to #3 obtained by the wideband interpolation process is carried out to remove aliasing components. Consequently, an image which includes signal components over the frequency band from −fs to fs and from which aliasing components are removed, that is, an image of a high resolution, is obtained.
As described above, according to the existing resolution enhancement process, an image obtained by processing an original image, that is, an input image, not by means of a LPF which passes a frequency band from −fs/2 to fs/2 of the Nyquist frequency fs/2 but by means of a wideband LPF which passes a frequency band twice the Nyquist frequency fs/2 is used to restore signal components exceeding the Nyquist frequency fs/2.
It is to be noted that, according to the existing resolution enhancement technique, in order to determine the weight for weighted addition with which aliasing components are reduced to zero, the number of frames to be used for weighted addition has to sometimes be determined in advance.
An image which becomes an object of a resolution enhancement technique is hereinafter referred to suitably as input image.